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legislaticn reguires that the Coiaptroller General of 
the U.S. develop and reconmend'tc the Congress methods for review a^d 
evaluation of government programs carried on under existing la«. This 
docunent is a first step in collecting and disseainatdng general, 
concepts on these activities and how they are related to other 
activities in the continuum of decisionmaking' about government 
programs. The document is offered., for the use of anyone who is 
evaluating programs and analyzing policy alternatives, in the sens§ of 
engaging in a careful/appraisal of what happened, why it happened, 
what choices are available for future actions, and what the 
implications are of/those choices. It is intended for uselby all 
petsons who are concerned with this process whatever the academic 
discipline or professional background from which- they approach the 
problem. While addressed primarily, to practitioners, it is hoped that 
this publication will also be a useful reference to tJioseLwho, as 
legislators or Managers, „ for example, are interested in the products 
of evciluation and analysis. The chapters discuss the framework within 
which these activities are performed and provide both conceptual and 
practical guidance. The emphasis throughout is not on advjanced 
quantitative technigues, bu;t on essential concepts and basic 
approaches. (BC) 
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For a number of years, we have been expanding the scope 
.of our work and the capability of our staff to^ review the 
results and effectiveness of Government programs. 

The Comptroller General is given very broad responsibil- 
ity' and authority under the Budget^ and Accounting. Act , 1921, 
and the Accounting and Auditing Act of 1950 for the evaluation 
and analysis of. Federal programs and activities. Section"204 
of the Legislative Reorganization Ac t of 1970 supplemented 
our authority and' ind icate'd current congressiona^l interest in 
analyses of programs.. The Congressional Budget Act of 1974, 
fur ther strengthened the congress ional emphasis on ^ the. evalu-. ' 
ation of programs, the statement of> legislative objectives^ 
and goals , . and improved methods of evaluation . Such wo2:k is , 
carried out .by all parts of GAO in connection with our general, 
review, evaluation , analysis, and audit functions* 

The 1974 act requires, among other things/ that "the 
Cpntptroller General " shall develop and .recommend to the. Con- 
gress methods for review and evaluation of government programs 
carried on under existing law.*'. This dofcument is > f irst 
step in collecting and .disseminating- general, concepts on ^ ^ 
these activities and " how they are related to other . act iviti^es 

-in the contin uum of decisionmaking about Governraent pro.qram_s.. 

The document adds to guidance contaihed^ in .Standards for 
Audit' of Governmental Organizations',; Pr^grains , Activities and 
Functions , issued m 19.72, which includes- in the full scope 
oF^sudh audits a review to- determine whether \desired results 
are. effectively achieved. \, 



The review and evaluation of programs and the analysis 
of alternatives are central J:p our mission. These activi^ties 
are also cr i tically " impor tan*t to effective management in 
executive agencies. •. An agency with. the capacity to assess 
the impact of its programs .and to examine alternative courses 
of action is much more, likely to pursue its program goals" 
effectively. One • dbj ec tive of this documen t , ther ef or e , is to 
encourage agencies to develop^-and use effectively — this 
capacity for critical self -assessment . 

'jPhis document is intended to be of value to the novice 
and the experienced practitioner whether engaged in financial 
audi-ts,. program, review, or in, program evaluation or analysis. 
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it is not a substitute for the many good textbooks available 
On particuyai: analytical methods. Rather, it establishes a 
conceptuar framework within which analysts from varying back- 
grourids can work and communicate effectively. 

■ We received numerous helpful comments and suggestions on 
'an" ear 1 ier >if af t o^ ' this document . We have taken those com^ 
ments interconsideration in pr epai: ing this final version. 



?e trust that this documt 
purpose, to encourage more ef 
anaylsis fn decisionmaking. 



will serve 
ceWive use of 



its intended 
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CijAPTER 1 ■ \ 

; ... • ; \ . ■■ 

* INT RODUCTIO N 

In recent years, a multitude of new programs have been 
launched to deal with the prob\lems of our society, and Govjern- 
ment expenditures have increased at a very rapid rate. Arie 
the programs working? Are the \fundS' being spent on these 
programs producing the results desired? .Are there better 
ways to attack and solve society ' s problems? And ^ are old, . 
programs achieving their objectives? I 

/Governments and agencies entrusted with puhlic resources 
and ihe authority for applying them have a re-sponsihility to .. 
answ/er these quest ionsTrto render a full accounting of their 
activities. Government managers have a responsibility to 
sho,W not only the purposes for which public resources were 
used^ but also to demonstrate the. effects, of their use.. 

...This responsibility rests first on the executive agency 
administering the program. Program managers need to know 
'what the programs entrusted . to ; them ard' accomplishing and 
, whether results might be improved. Ceiitral management agen- : . . 
cies— the Off ice, of Management and Budget. or the; White , House , 
for example, need to know if programs are ..working-^-eith'er through 
their Vowriv efforts or through the review of the, appraisals of 
managing"' agencies. Finally r congre^ssional responsibility for 
legislation, appropriations, budgeting and priorities, and T 
oversight and investigations indicates a need for the legisla- | 
tive branch to make its own appraisals of .programs and to « 

make use of appraisals made by the executive branch. 

■ . . ■ . • ■ *^ . • 

The ultimate choices about programs — ■decisions about 
whether to do or not to do something--will bepolicy choices. 
However, political leadlsrs, public administrators, and the . 
public need as much information as. possible on the^ choices 
that must be made. ^ This need has stimulated the deivelopment 
of various analytical techniques which have been grouped 
under labels such as program evaluation and policy analysis. 
The art of evaluation and analysis is not yet sufficiently 
. developed to permit preparation of a manual coverings "how 
to do it" in every situation. I't is only evolving. While 
recognizing this, decisions must and will be made byjlegis- 
lators and executives faced with the task of formulating and. 
reformulating programs to deal with , the problems pf our.. 
-society. . " ■ • ' V- ' \ i; 

■ Thus, we offer this. document as a:' first step in collecting 



and. disseminating lessons learned in ,GAO and elsewhere about; 
analysis and evaluation- Generally speaking , w? of fer this \ 
guidance for the use of anyone who is " evaluating ". pr^ograms 
and "analyzing" policy choices in the sense of;engaging m 
a careful appraisal of what happened ,. why it happened, . what 
choices are available for future actions , and what the impl i- 
cations are of those choices. We intend it for use by all 
persons who are concerned with this process whatever the 
academic discipline or professional .background from which 
they approach the problem. While this statement is addressed 
primarily to practitioners, w'e hope it will also be a useful ^ 
reference document to those who, as legislators or managers, > 
for, example, are interested in the;products of evaluation and , 
- analysis . - 

The concepts and guidance which we offer, must be/adapted, 
to specific program situations. Program objectives, are seldom- 
as clearly stated or agreed upon as would be desirable for 
evaluative purposes; no program operates in isolation from 
other social or economic events; and data and measurement 
techniques are almost always less adequate than desired. 
It is in the adaptation of the ideal and the theory to the 
specific sftuation that the persons doing the work show their., 
worth. The judgments involved in the identification of ob- 
jectives ,1' the selection of data and measurement techniques, 
and the evaluation of external factors transcend in importance 
any "rules", which can be prescribed in a docum,ent of this 
sort . ■■ , .. ," ' ■ 

The chapters which follow discuss the framework within 
which these activities are performed and provide both concep- 
tual and practical guidance. The emphasis/throughout is noti- 
on advanced quafft itat ive techniques, but on essential concepts 



and basic approi^ches. 



CHAPTER 2 



THE PUBLIC DECISIONMAKING- PROC ESS 

^" ' . . . 

There are many reasons f or\ governmental activities..^ 

For example , .national defense is provided ^ecause there is 
no suitable private,^ alternative . The pro\/lsion of free public 
education reflects society's preferences and its .notions about 
its own long--term welface. Health and, safety considerations 
lead to the regulation of pr ivate activities to avoid adversely 
affecting the public's well-being. Tax incentives such as 
•the investment tax credit are intended to affect" the per- 
formance Vand growth of the economy. The relevance and ^, 
effectiveness of any governmental' activity may be questioned.^ 
Analysis, evaluation, and rela^^ed activities are tools to. help 
decisionmakers in resolving the^se questions. 

The private marketplace has limitations, particularly 
because it does not produce 'certain goods with high social 
value. It does not always provide adequate information., 
sufficient' competition, efficien^ designs or qualities of 
certain goods, desirable distribution of income and wealth, or 
desirable modifications of consumlption patterns. But, re- 
sorting to' governmental action because of these f ailingsdo^s-^ 
not automatically insure that thei same or other failins-s-^ . 
will not occur. The effectiveness and ef f ici.ency^wrth which 
the Government performs its functi^pns must^al-so be weighed' 
in deciding whether governmental acti9.n--should be enhanced, , 
changed, or in some cases., is warranted at all. 

Each year. as legislative, budgetary, and appropriations 
decisions are being considered, the pr^actical issue remains: 

-W hat ■-do e s ■ -th e, ■ p u b 1 ic^.. n eed .-..an d ... h p. w- . '.s h o u i:d....p r.i.D.*r . it ,i.e, s .i^be. — : 

established? ' In a democracy , the political process is relied 
upon to examine and determine public need^arid to set priori- 
ties as to how such needs are to be met fr.om public funds. 
Elected officials are responsible for learning and reflecting 
their constituents ' needs and proposing programs or program 
changes with requisite funding levels to assist in determining 
priorities for action. \ " 

But. for the work of elected officials to have, meaning , 
accurate and relevant information must be available and , 
useful debate must take place. Analysis and evaluation help ■ ■ 
prbvide the needed infotmation and a basis for judgment of 
those, persons and groups involved in public decisionmaking. . 



THE RESOURCE ALLOCATION /PROBLEM 

Government actions generate bknefits and \incur. costs. 
These benefits and costs sho'uld be I broadly defined to include 
their social and private aspects, \The Key elernents of the 
pr„oblem of choice are: 

—Government objectives are achieved by. developing _ 
adopting, and., implementihg policies, and by creating 
and- operating programs , all 6f which consume or 
transfer resources— tangible land intangible. 



— There are many public needs, '\ These needs are large 
and constantly changing. Demands for resources 
are much' greater than the resources av^ailable. 



— Decisionmakers must choose among competing objectives 
--'"and among the alternative programs and \policies capable 
of meeting the\chosen objectives at desired and 
j affordable levels of achievement. \ 

Thias, decisionmakers are involved in the process of 
allocating available resources among competing demands so 
as to achieve the greatest overall level of net benefits, 
possible. At the same time, full consideration is given to 
justice, equity, and political reality. .r 



IS SUES IN, RESOURCE ALLOCATION . „ | 

Most programs are interdependent and affect more than 
one of society's goals. This leads to a need for use of 
evaluation and analysis in two resource allocation Ussues : 

(l) choices ,with,in,:..,a major program air ea a (2) choices among 

major program areas. 

For choices within a program area, the following 
quest ions ■ are often posed: ; 

— What is the appropriate level of attainment 
for a given objective? 

— Are there prefer red . a,lternatives for reaching that 
level? .. ^ . 

. — What are the resource requirements for doing so? 



■ — Are there obstacles to acdeptance and implementation 
of an otherwise preferred/alternative and what would 
be the costs of overcominn the obstacles? ; 

— Are there equity considerations connected • with ^ 
, the leading alternative^^ * 

For choices among major_.program areas similar questions 
_ace- relevant . Judgments '^'that are made by decisionmakers', 
concerning relative/lmpor tancej of the various objectives will 
affect the assignment of resou'rces among those objective's , 

Resolution of .resource allocation issues may result in 
any of a number of actions: (i)(a) continue, modify, or ^ . • 
abandon existing policies and (\b) "adopt ^new polices or ' 
(2) (a) continue, modify, expand\, reduce, or phase out current 
programs and. (b) create new programs. 

The decisionmaking process and the relationships am,ong 
its component functions do not n'ecefesarily follow a 
predetermined sequence. The decision process and the ^ 
supporting functions of evaluation and analysis are concurrent, 
continuous processes, wi th continuln'g interaction among the 
various parts. ^ -y 

"' •The role of evaluation and analysis in* the allocation . - 
of resources is to provide bettet\informat ion., about the . 
implications of the choices available' to the decisionmaker.. 
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CHAPTER_3 '7 ■■ ' 

; ■ . THE EVALUATION' AND ANALYSI S ^j60NTINU UM ; ' . . 

' Evaluation and analysis covers a wide range /of activities 
designed to support the ongoing decisionmaking process. . 
These activities include r-eviews known as program auditing,^ 
budget examination, management analysis, planning , institutional 
research, program budgeting, systems analysis, engineering 
economics analysis , program evaluation , policy ana.1ysis, 
cost-benefit analysis, etp. . . • ; ! i 

The specific type of review needed depends, in ij)prt, 
on the particular issues' raised or questions asked ai^d the 
focus of the inquiry on the part of the* decisionmaker:!. . If .„ 
financial inforination is needed about a Government agpncy or 
corporation, an audit of the results of pperation and| financial, 
condition for a given period may be; the appropriate fo.rm of 
review or analysis needed. If there is concern about] the . . 
management of. a program, a management review that isldirected - 
more towards the effectiveness of the organization, management, 
and staffing may be appropr iat^e . j ., 

i ■ ■ , * ■ ■ ' ' . ' 

General program reviews that look at program effectiveness, 
and consequences, as well as management effectiveness /are 
useful to the decisionmaker Iq determining whether the program 
is meeting the established objectives and whether thejre are 
changes needed to improve the program efficiency and jsffective-- 
ness. A comprehensive review of sir\ existing program may 
consider the overall performance of the program, incl|Uding an 
evaluation and analysis of performance on any number.of 
criteria. Consideration of alternatives may also be |included. 

■ ■ ■ ■ , ,.• ■ . ■ '1.. ■ ■ ■■ 

I t should be recognized - that trad i tional per ceptdons ..of . - 

terms such as "evaluatiofi" and "analysis" tend to overlap^. :In 
some cases, the two terms are used, interchangeably. Other . 
terms, such as "auditing" or "planning," may also be used 
interchangeably with evaluation and analysis. . 

, CONCUR RENT P ROCESSES IN THE CONTINU UM . ■ .' " 

For purposes of this document, drawing sharp distinctions, 
between evalQation and analysis is less useful than focusing, 
"on the two basic questiohs which decisionmakers, and. their"- 
staffs, face: (.l)'what actually has happened as a result; ; 
of past or current policies arid programs and what have 
we learned? and (2) What .should be done in. the f utur e and ' - ;, v . ■ 



^s^^wh^t ar.e our options? Answering these questions can,., in, ,, 
turn> be cougnly translated into , broad classes of act iv.ities : 
appraising the results of policies and programs and assessing, 
alternative policies and programs. These broad activities 
include the variety of review types listed previously. 

An example of the interaction of these concur r^nf 
processes in ^-^^^^^inuum is shown in the follov^-ntl 

illustrat - 

Aft ^ro^ is identified, the^^ir - s':^p snou^d oe . 

a systemat jiv..empt-to assess what caused Ue problem and 
what the alternative , ways are^ to solve it. Alternative _ 
approaches , developed as part of tne assessment, do nqt- 
normally contain the level of detail that would be needed. for 
actual implementationi ' For example , after the decision is 
1 made to implement ' a cprogram , the - initiation of the program 
. Would require a detailed plan tnat specified sojne" or all 
of the following: processes and technologies to be employed, 
investm^nt.and operating funds needed, and capital facilities 
. needed together with the acquisition scnedule that will 
provide -for the scale of operations desired , within tne 
apprbpriate timeframe. 

" It .is important to build into the implementation plan 
^ ' specific provi.sipns for g'atheibing information necessary for 
aicomrL'eherisive arid valid appraisal of results., During the 
implementation phase, such information should include , where 
possible, tne/^effect^ of any changes in thje implementation plan 

' " . As. required to suppbrt decisidnmaking , . f ur ther assessment 
* of alternatives should take place concurrently wi.th appraisal^ " 
•of .results during the operation of the program. Appraisal of 
results without such assessment provides only limited Insights 
"' since it. does not address the desirability of new alternatives. 
Conversely /-vassessment of alternatives without sound appraisal, 
of results lacks the insights which wpuld come!v^jL,|^^^ 
derstanding of the existing program. A high . d'dgr ee of ' interac- 
■tion exists not only, within the various phases .of an assessment 
or appraisal, Dut between the two processes. If these inter- 
actions and feedbacks are ignored, the quality of both efforts 
'Suffers . ■ ' 

. Tne combination of appraising results and assessing 
alternatives shpuld answer the following questibns : 

— What was the impact of the existing program? 

Could a comparable level of results, effectiveness, 
or Denef its be achieved a^ lesser cost;. 
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or for the same costs could greater effectiveness, 
or benefits/ be realized? . * \ 

— Could these more desirable outcomes be achieved - 
by reshaping or red irec ting .\the current program or 
policy / -adopting some . previously* suggested*, but / 
rejected, alterna tive ; substituting some wholly .,, \ 
new alternative; or creating ^some combination of 
. the foregoing possibil ities? \ 

■ ■ _ \ . ■ . 

Real world decisionmaking and ma'|iy of the activities ; 
which support , it are complex , uncer tain , hurried, and 
subject to all kinds of cons tr a ints : \some under standable ' 
and some seemingly arbitrary. These d^if f icul ties are . 

omppunded by the ex ' tence ,of competing or complimentary 
objectives ( or their (related programs ) which' may 'also- have 
CO /be considered . Evaluatorsy- analysts , and other 
reviewers can do little about these difficulties;, but/, if 
they understand the complexity . of . the situation , they* can 
perform in ^ way which *is of .maximum usefulness under . 
the circumstances./ 

ROLES PEOPL E. PL AY V . \ 

• The distinction between the cojicur.rent activities on 
the. one hand and the people or organizations that perform 
..them on' the.^other^^ hand is an additional source of confusion. 
As noted earlier^ appraising policy and program results 
and 'Assessing al terna ti Vf/tS are related and mutually reinforcing 
processes. These proc^^jrS<5es can be., and, fr^equent^y ate,, 
performed within a singl'^ organization' and of tern, fay ttee ' , 
same person^ This has significant advantages^ of efficiency 

and of ^-keeping the pragrtt ioners Of the various skills aware of 
useful interactions. 

In the real world ^ation a variety of peiDpIe are, 
involved who have differ nt backgrounds and call themselves 
by var ious 'titles.. Many of these people *move .fcieJ!r.ween . . 
appraising- the results of policies and programs ^nd assessing 
the alterriatives for improved choices in the future. At . 
times^ these people may be emphasizing the assessment of 
alternatives, at other times the appraisal of resul ts.,..;and 
sometimes both together. People who gain\ their first 
experience in one activity move throughout the cr>ntinuum _ - 
and interact with ot'heir people having other exper ience when 
working on a particular study. ^ , 



No gro^up" <^ii profession has a monopoly on the-talents 
required of a good evaluator or analyst. The basic 
prerequisites are (1) an irvquiring, skeptical, challenging, 
mind, (2) the ability to think systematically and ^ r igor ously 
■ and. (3) an openness to new ideas. This "mind set" ob- 
viously needs to be coupled with an appreciation of the ^ 
usfes, powers, and limitations of such" fields as ecojiomics, 
-statistics, accounting, operations research, etc. When ■ . 
hi4h' levels of skill in these and other areas are required, 
the practicing evaluator, analyst, ;or other reviewer should 
i-ecognize this fact and call on th-ei needed expert. 

,-,The following chapters fociis on those ' ideas , concepts , 
?and approaches which are. basic to appraisals of,, pol icy and 
/program results and to more ijisightful assessments of 
alternatives for improvements'. ■ ' 



■ ' ■ CHAPTER 4 ' ' ■ 

- ■* APPRAISING T HE RESULTS OF PQLICrES;_^AND^_PRgG^^^ ^. 

■ - ^ ' " AND ASSESSING ALTE RNATIVE APP ROACHES 

The previous chapters discussed problems and issues in 
public decisijpnmaking with par ticular emphasis on resource 
allocation and on th^ continuum of evaluation , analysis , and 
other review functions which 'support dec isionmaking • , T^^.^ 
chapter discusses the methods and concepts associated with 
/ • evaluation and ^ analysis . , ; •■ ■ . . . 

■ ..A >i--~Th.e discussion that follows focuses on : " — — 

^^^^^^^^^ --understand ing fundame^htals in appraising :resu^^ ; 

• ; ' arid assess ing alternatives \^of * programs and policies, 

V ■ ' — ^'appraising policy and progr^Ti"^ results , v and 

a'.:. -; . --assesstnij^;^olicy and pr 

Obviously the degree to which, the. me thods can .be applied in - 
a par tic,ular case depends on the specific problems to 
: ' be considered.' It is not always necessary to use each 

! ' ^ - of these methods in the order presented heire. 

.:*'.•* ' " . •. . ■ - ■ ir'i- ■ ■ " *^ ' ' 

.■■V ■ ■■ ■ . .. ■ • ... ■\ 

DNDERSTA NDING FUNDAMENTALS ^ 

. , The activities of appraising results and assessing 
r . . ' . alternatives^ of programs and policies s certain 

• * ' . • fundamental concepts in which the mode of inquiry is 
iH'^^..^^- essentially the same. These fundamental concepts include 

\ ' —ascertaining decisionmakers' heeds, ^ 

. ' . — defining ^^the nature and scope ot the problem, 

7 ^ : '—determining valid objectives, and , ^. • 

* ' • — ^specifying: comprehensive measures. 

C,' : Ascertaining e c i s i o h m a k e r s ' ne e d s 

■ . An 'initial task- in either 'appraising results or assessing 

i • alternative^ is to develop a clear understanding of the 

decisionmaking. needs. These needs can generally be summarized 
by. answering a series oE questions. ■ " . 
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—What is the decisionmaker's perception of the problem? 

— Is there dissatisfaction with, effectiveness or 
consequences of the policy or, program? With the 
lack of a policy or program? 

—What uses are to be made of the information to be 
collected? " 

i^^ .. —When is the final report needed? Will interim _ 
- reports be needed? ^\ 

It is helpful^to make some distinctions among the varl-jus 
par t ic ipan ts in the ~de^ is ionmak ing- process . 'Some may - al read-K^- 
have an understanding- of the 'hature -of the problem they face, 
what they want 'to,, know , and why.. ~~Oth^- mnv obI y have a 
general percept lorrV of the problem and Wi.ut needs to be done 
about it. rn the la'iLter case, a more extensive-discussion 
of these fundamentals may be needed to develop, the basis.^for 
a study that will be useful to these decisionmakers. The -^.v^ 
nature of . the issue will have a major impact on the sort of „ , 
information that must be collected and the sort of ^analysis 
Chat must.be done. Some environmental quest ions for example , • 
are 'not as'^susceptible . to . prec ise" answers as would-be . desired - 
by the dec is lonmaker . • ' 

It is important to recognize^ di^fferent (and poss ib;Ly4:' ^ 
conflicting) viewpoints and, in.te rests among par t ic ipants • 
in the decisionmaking process. The of f ic/ial ...sponsor may -be, 
a congress'ibnal committee, whereag.. the;, real useiTi.o.f the study v 
may be one 'member of ' th§ committee .or .the committee staff. 
Other participants in the process inclucSe the managers, of ; 
the prSg ram' being evaluated or analyzed"; >s well as managers / . 
of related ' programs. Another ..congressional committee , ^ the 
Office of Management and Budget, or a, pr ivate organization ,, ■ 
such- as an associ,ation or Government contractor, may become^- 
increasingly interested as the' study progresses.. • - 

In developing a clearer, idea of any of . these participants ' 
needsv attempts should ' "be made to elicit and ^^.^^f JJ^^ 

tion on the nature of the problem or' issue as it is curre|^tly . 
understobd, the general context of the P'^°51em, .and part|_ 
that appear to require special emphasis. Spec if ic attention 
should be given to the order of priority in approaching: ±he , 
various parts of, the problem and to particular points o:f 
information or insight essential to Jaking the . decision or 
meeting the decisionmaker's needs.. The bureaucratic or ^poli- 
tical coni^xt in which the decision will be m^de needs to be 

", . . •..■■'■,19 ■■' ■ ■ ■ ■■ 
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understood. The time ava ilable ' f or the study and the critical 
points at, which specific items of information are needed ^ 
should be ascertained. 

Definincj the nature and scope of the problem 

it is essential that the intended users of the, study 
(the participan.t.s in. the decisionmaking process ), together 
with those persons responsible for the studyr share a common 
understanding of the nature 'and scope of the issues at stake. 

A full anc3 correct understanding of the nature ^'^<^ 
problem will be, aided by (D considering its origtr,, ..aown, 
(2) reviewing IcgislaLive i .;ings, reports, and acts 
associated with it, (3) inquiring into the history of programs 
'designed , to, deal with the problem, and ( 4 ) . examining pas* 
analyses, evaluations, audits, and budget examinations o€ the . 
same: or related issues. Outlines- and checklists can be' 
heipful in this review.. ^ , 

The scope of a study depends both on the questions it 
waaald.be desirable to answer and the availability of methods 
.and data which will provide those 'answers There, is of ten 
at tradeoff between -the- breadth of a study and precision in. , . 
ttre results. The planned scope 'should also consider the' 
na^ture 6f the decisions which the study, may, affec.t,. For , . 
^example, a broa-d consensus that a program, is.t^no^fe working , may 
warrant paying more a't'tention to ba^ic'.altetJi^.eives and less 
to minoY'improvemeriJb.S: in the existing programV A shared • . 
understanding o'f^ the .scope of the study and the objsectives 
and-aneasures of the, policy -or_prog ram are the foUnda.tiDn for 
defining, -the 'Initial dir ect ion ~^of;-the:v study effort. 

Similarly,^ understanding is needed of the coverage,,^ , ,, , 
required in geographical terms . (regional , State, local); on 
areas, populations, individuals .or units to be included; and^ 
on :the scope ( how many indlv iduals , approximately . inow much 
ii,nfoxmatibn : from each, etc.). The scope of coverage together 
..with the timing provides for the logistics, of the work. For 
example, indepth . work'may be . under taken .at a small iraiimber of . 
locations, less detailed work at a larger number of locations, 
or some combination of these. 

. Determining valid' objectives • . " 

, ■•: Tihe objectives' of the policy or program^-the. benefits 

desired to be achieved — frequently are not stated clearly 

and precisely. The original sponsors of the policy or program 



may not have had a precise .Ade.a-,LOi....th.e. end results desired. 
Formal statements of ,;6bj ectives' may . be intentionally • ambig- ' 
uous if it is easier to obtain '^O'^sensus on action. Value, 
judgments underlying, the objecti\es may not be, shared by. 
■-important groujis'. .. Cbnsequently, \he end results intended 
may'bepeir'ceived by '^some as implying ill- effects for them. 
Furthermore', explicit sWtelifrents of \^bje°c.tives: tend to imply 
a specific assignment of priorities and commitment of re- 
sources . . ■ 

^ To the extent feasible, t.a cement-^ of objectives should: 

. 1. Capture a complete under stand ing of the intended 

* benefits, including the expected level of attainment.' 

Identify nscipients of unavpidable adverse conse-' 
quences or unintended benefits. 

3.. Include i35)0'rtan€^' qualitative aspects, aveii though 
measuring degrees of. attainment may be exceedingly 
difficult. /; 

•4. Take account of. mult iple , objectives which may be 

complementary or conflicting. . ',. -■ . 

The importance of taking such a comprehensive view of 
objectives cannot be overstated., ..Oversimplified statements 
(1) will not capture ail , essential aspects of the effects 
intended and (2) may contain implied conflicting consequences 
for groups other than the intended beneficiaries (e.g..-, to 
e'liminate' hunger or to achieve energy .sufficiency ). . Implied 
objectives may representi'des irable end results . For example , 
a summer employment 'program aimed primarily at increasing 
earnings of young people, may be viewed as reducing civile- 
disorders. Moreover, even desired end results may not all ^ 
be . achievable simultaneously and may be xnterdependent. 

Oversimplified '■statements may result if activity .jrtile- 
stones are contained- in them (e.g. , to. increase the.i number 
of emergency rooms by 20 percent by ,1978.) . An objective ; - 
stated in this way may ovetly constrain an assessment of 
alternatives,-. the purpose of. which is to determine-efficient 
levels -of - attainment, of .;an ultimate -benef i t .— On thje other ■ 
hand", it is important that the statement be specific with 
rgspeqt to the nature and . direct ion of change so that .progress 
can ii,e"'measured.' A statement such as " to reduce deaths , 
additional complications, disability, and suffering:; of persons 
with acute' injur ie-s by irrproving emergency care'" would satisfy 



these criteria. Quantitative goals or. targets are also 
needed, bu.t these muot reflect priorities among programs. 
Accordingly, they can best be set as part of the budget and 
long range financial planning process and should be reexamined 
regularly as budo-t priorities shift.' 

In. appraisir suits of ou'joing programs, if targets or 
act Ivity milestones have been furnished to managers, the 
targets or milestones should not be accepted uncritically 
(e.g., a specified student/teacher ratio ) . An attempt should 
be made to find whether deficiencies in attaining the mile- 
stones are caused by unrealistic expectations or by the way , 
the program was implemented or operated. 

A shift in objectives can accur over time and care must 
be taken to assure that statements of objectives currently in 
use are still valid. For example, the objective associated 
with the national 55 miles per hour speed limit, now includes 
safety, as well as "energy conservation. . 

• Determining valid objectives is a complex and frustrating 
task. 'A study may have to proceed without fully satisfying 
.these requirements; I'f this is .the case, objectives should 
be reexamined and clarified as the study progresses. 

Specifying comprehensive measures : ^, ' 

Valid measures of policy and .program consequences are re- 
quired for both appraising results and assessing alternatives. 
Objectives and measures of consequences ar e Interdependent . 
The quality 'of each.depends on the other . " Measures should 
be used which cover all aspects: of a given objective. Ideally, 
. measures should ' ' ■ ! ' , 

—quantify the -extent to which' the objective(s) are or • 
would be met — "effectiveness" measures'; ■ 

—capture quaLitative' aspects of the consequences— 
"intangible" measures; ,. , 

■ ■ . ■ ■ ; :. ■ ■ ■ ■■ . ■i'-'v, ■ ^ . ; . . ' . . ■ v ' '■ - 

--quantify, to the 'extent possible, unintended consqr 

• quences-.-"sixae-ef feet" meas'ures ; ' ■ 

—quantify,^ to the extent possible, the differences O^Br^-- 
impact on the beneficiaries and the cost bear^ers— . ■ 
■'d is'tribiUt ion'"- measures . ■ ■ .-' .~ •'^'^ . . ■ ' 

When appraising .results\ it may be* decided for: practical I 



reasons to exclude side-effect, and distribution measures.. 
However, they should not be excluded ijE reasonable effort 
would oroduce useful data or if the decisionmaker is espe- 
ciallyVintprested in these measures;. For intangible measures, 
some qualitative, indication of relative'-magnitude should be 
used (e.g ., ratings by clients reflecting, their satisfaction 
with the quality of a service). 

• Data may not be "available on the desired measures or., , 
if available, may be obtainable only at high cost. In these 
cases, surrogates will have to be used. For example, the 
scholastic aptitude test is used to measure likely achieve- ■ 
-ment in cpLlege. When surrogates are used , their validity 
should be established. 

There is a temptation to define quantifiable measures, 
especially of effectiveness., too rigidly or narrowly. For 
example, in "evaluating a public employment program, a success- 
ful participant might be specif ie.d as a person who is employed 
1 year after compttetion of training. If the participant worked 
1 day-le-ss than a year, would- he or she.^be viewed unsuccess- 
ful? Suppose, the participant only occasionally held a Dob,_ • 
but happened to be working a year after the program. Should 
this be counted as a success? The 'range and distribution , 
of outcomes should be cons ider ed„ in th is case . For example, 
data on the percentage of per sons -'hold ing -jobs .for -various r ^ 
lengths. of time after trainiihg wou;id provide a more meaningtul 
picture of real outcomes. This sort of distributipn is 
needea to' j udge ' level^s of a tta inment or degrees ^of success 
or failure. , 

■ . These four fundamental concepts are closely, interrelated;. 
A clear understanding of 'what is . needed for the decisionmaking 
process, of-.the naturef of the problem, and statements of 
objectives is necessary in order to assure that a meaningful ^ 
and feasible set of rneasures; has;' been specif ied., 

APPRAISING POLICY. AND PROGRAM RESULTS . V 

The process of . appra is ing results should begin concur- 
rently with policy or progr^am impTeroentation and continue as 
needed. Continuous appraisal,, through a well structured 
management information system, should be mamta med , , but .. 
•even when it exists there will be a need for special reviews 
from time -to time. ,, ■ ^ . • ■ 

After the fundamental concepts discussed above are ■ 
'understood they must be fur theri developed through application 
' .of other more specif ic conjceptsj and methods including .. - 



—making valid comparisons^ 

— developing needed information, 



^ . — interpreting program results , and 

—checking the completeness of the appraisal. 

' ' ■ • ... ■ ' * ■ • . 

Making v.alid c ompar isons 

Comparison is the essence of appraising program results. 
i'or evaluatibn to be useful, outcomes of the program or policy 
must be compared with spmething else in order to reveal the 
effect of the program. The basis for compar ison may be . the 
•outcome of an alternative policy, of the same program at , an 
. ear 1 ier • time , or a combination of such compar isons. , 

The measures that are used in making these compar isons ' 
should ber^derlved f rom a val id statement of objectives . This 
derivation 'is moreSdif f icult if the objectives are not clearjy 
. statedV changes are takirig place, or for other reasons the . 
basis of comparison is arbitrary. - 

The measures' used for comparison need to |be: developed 
with the decisionmakers' who know wh'dther their focus; is on 
resource input ('e .g mixture of par aprof ess ionals , nurses , / ; 
and physicians) , operational process ( e .g . , scheduling of <r 
surgeries), outcoih'es (e.g., disability .days^-'ayer ted ) r the 
operational setting (e.g., interaction with other .outpatient • 
and inpatient facilities)"; or soniQ combination of , these . • 

> •':Other sor;t.s ofJ'com|)^^ useful For example , ^ ^ 

comparison of m va:r'ia.t ions . among^-pro jects within an 

"existing progr^'r^'^^^^^^^^ to identify important characteristics 

and potential improvements. A comparison of similar programs, 
if feasible-,'- may provide . some of- the same inf ormatiion,. 

:V - Once the rvature o'f . comparisons, is established , a series 
of additional questions directly relating to the .problem at 
hand should be^ raised. ' Some of these" are posed as; hypotheses . 
which the- appraisal aims to prpve or disprove^ Siich hypotheses 
""or questions must'allow an appraisal of whether consequences . 
or effects are attributable to the program or to "some other ' 
causes; A decision needs to be,. made whether only jdes.criptive 
•findings, will suffice or whether it .wil 1 be necessafy \to 
demonstrate significance of . results, or differences in effects. 

Choice of ;a compar ison approach depends both upon the .. 



questions to be asked and thQ availability of.data. This is 
not only procedural ^ut involves also questions of access,.^ 
comparability, restrictions on collection and use of..confr\" 
dential data, etc.-' These problems may be more severe than 
maiiy evaluators, auditors, examiners, and others realize. 
This means that analytical methods should be u^ed which make 
the most • efficient use of each ^Jp^t of da ta . 

■ Data problems may make it impossible to use the best ' 
theoretical method. Some methods may be impractical if data : 
is too highly .aggregated, incomplete or missing , or may re- 
quire ''patch up'Vefforts alter the evaluation^ is underway. 

Some major comparison methods follow. 

Experimental methods — attempt, to measure. the results of , 
the program as though everything else is held constant. This 
is . done by measuringv-the difference, in ter.ms of ^ the measures, 
of success,.- between -those affected by. the program .-,^nd a cdn--- 
troi grouD" which is not-. This is the preferred m.ethod for 
evaluation of social exper iments , . but ' i t can also be used for 
any evaluation -when the essential requirementa-of random 'assj-gn 
ment .and control are feasible.. This is the approach; that was 
used in the New Jer'Sey Negative Income. Tax experiment. • In ;^ 
that ^experiment, several, d if ferent "amounts of monetary ^ 
incentives, wore .given to'*dif f erent gtoup^' of families, in the^.. 
^same situation to see what effect the incentives had on .work' 
and spending' habits. Responses were compared with the habits 
of families in the- same situations which received no. monetary 
incentive from the experiment .during the same' time . 

■-.The analytical strength of the experimental .method mak'^s 
it a very useful tool.- This value must be balanced: against 
other considerations, . such as^i cbst and ethical , and - legal 
constraints, before this approach . is selected ; • • 

. . • Experimental designs •'t.egulre tfi'at, the affected^ group arid 
the group, nbt affected possess similar; ; characteristics . This, 
is the reason for a ,str ic.t ' requirement bhat the potential,. . 
par ticipants ' be- random.Iy assigned so .that; each ■ one has=' the . 
same chance of assignment' to either • group. '-before ■ t^ prog.ram 
begins. Unless randomization. Is achieved., , there is no assur- 
ance that the -results.,. areV-:attributabre^,t^^^ 

For 'example, unless 'randomry assigned", par.ticipants • 
might enroll "because they are more percep.tive and desire the 
benefits more than others who .rare >li^ ible . This biases any 



comparison of the response .'or performance of the two groups 
because their moti.vations and other characteristics were not 
the. same . x • ' ; 

Nonrandom compar ison_gr^tJp__methods--ar e commonly used . 
when thF requlr ementifor strTct randomized control cannot, be • 
satisfied. Attempts are made to make, the comparison group as ■ 
similar to the experimental group as . possible by matching ; • • 
individuals with the- same sex, age , rac ial , ' or ' soc lo-econom.ic . 
characteristics. ..'The differences in results .between the^ two 
groups (experimental and the matched comparison) are attri- 
butable , as , in experimental designs, to the results of the . . •, 
program. However, without random ass ignmemt '.there .is greater 
danger that the observed . results are attributable „to . nonprogram 
influences. - Other difficulties, with the method include poten- ; 
fCaf.'Si^as resulting from self-selection by par tic ipants . • 

Co mpariso n of .similar ,p£ograms~attempts to establish 
measures and data with wFTch theviDutcomes of two or more 
■vbngoing programs or components can be compared. .Progranv . /• - 
compar-^isons- are .attractive for;several reasons,.; . They; (i)^- .. , 
providfe informa'tion on effectivenesss. of alter nat'ive;s in .coiji- 
parable terms. Ind for the sanie time period>;r(2) ^reducevthe 
need to rely -cortipletely , on ^.the elusive ./'cori^trol' of exper i- .■; 
"mental me.thods applied' to .onfe .p'l^bjecf, (3) ihelp generaLr^^^^^ 
: the resui'ts if widely' distributed " representative,'' proq.ectfe 
can be included, and {4) offer an opportunity to identify \ 
exceptio'nal performance arid" to study what is;.,oper atipnally 
.different, about those ^proj ect.s.^^^^ ' . 

Program compa-risons coiriprehensive enougt) to 'yield the 
above.advantages .are costly and difficult to. rti^nage. For 
example, although ' "planned . var iations,''^^ must be carefully^ ^ , 

• docbmented at, the outset, -once in operation . they will seldom 
tie free of ' fur ther changes , which also ■ must be documented . 

' it -should be noted whether such changes are ."pbsitive" . 

..(efforts to apply even better, fiiethods) or "negative (resis- 
kance to Adopting the prescr i,be!a methods ).. ' , " . . ' • ^ 

f Time series— involves a series of .measurements at peri- . 
bdic intervals before the program begins, and during .the 
fprogram. For. example , in evaluating the- safety results of,. - 
Connecticut's crackdown on speeding, it was possible to .i^se 
time series data collected for several years before and af^ter 
this new policy change. An abrupt change in such_ trend data 
is strong evidence ..that the action taken caused the observed , 
change in b'he trend. If measurements can also' be obtained 
in another setting treated as a compar ison group, additional. 

■ insights are possible. ' ^ ' ■ 



Careful interpretation is needed when using time series 
data. There" may be a time lag between receipt of services' 
and* the impact of the services. The analyst should also be 
alert for cyclipaa phenomena, such as unemployment Iqvels, 
which might cause part-cf the trehd. 

* . 

The methods discussed above are not exhaustive and there 
'are other ways -^bf ;iiaKing useful comparisons., 

Developing needed information ' 

Many information systems are not structured to capture 
data necessary for making comprehensive and valid, appraisals, 
consequently r a certain amount of ad hoc data collection will 
be necessary. . Repeated'appraisals of the same programs will 
be 'aided by incorporating' procedures,^to capture the desired , - 
data on a continuing basis. For new programs, a special effort 
should be made at the beginning to incorporate specific pro- 
visions for gathering the necessary information. In any case,, 
decisions are required on ^ ^ -r-.'. 

, — precisely what que.stiohs are to be answered and : 

— specific, items of'data required for analytical . methods 
' to be ' employed . * % 

Selection^ design, and implementation ^of data collection 
instruments may be' the least attractive aspects of any ap- ; 
praisal , but they are among ^^the most impor tant . Major- sour ces 
.of data . include 

— interviews, 

— mailed . questionnaires , , 

• : — onsite observations, 

— peer-:-group ratings, ^ 

^-•^standardized written tests,! ' ^""^ ' • ; 

• — project and other pr'ogram records, 

--Federal and State "government statistics , such as those 
from^€he Census Bureau and Bureau of Labor Statistics, 

--performance tests 'or other, physical evidence, 



— ^clinical examination, 

--financial, cost accounting, and operational manage- 
ment information, and j. 

' ■ . • i , . ' .- ' 

— documents such as minutes, progress reports ,.. public 
releases, etc. > i - 

Usually, it is helpful to use sever al sources , and there are; 
opportunities f gr creativ^ity in^ design of collection instru-; 
mentsand in analytical designs which merge data from several 
sources. It may be- very helpful tqlmerge data obtained from ' 
personal interviews (condition of horne , etc . ) with^data in 
program files ( achievement scores, e£c .). Careful design, 
in a technical sense, must be coupled with, careful cbnsidera-. 
tion of preserving the confidentiality of data about indivi^ . 
'duals. (See. ch> 5.) j , ; . • 

Inter p reting program results ; ^ 

The key point in interpreting the data is to ascertain • ■ 
the degree to which results , ^consequences or effects * are 
attributable to th§ progr:am( s ) . or to other -external influences. 
•Frequeh^ly^' the data will. ■ reveal only small effects . Even 
small effects are important, however, becaus^ they may be the- 
only clue available to the potential for larger^ e.f f ects/wh ich 
were either obscured in the data or are achievable only 
through greater change, in the program. Because of the poten- 
tial, for large effqcts to be obscured by the 'data, it is 
important to examine small effects v^ry ' carefully . 

Even if valid, dependable results are not obtained, the 
data should provide insight into the 'structure needed in 
'^litther research and evaluation. These insights may also 
suggest new 'and different alternatives which should be i , 

assessed . I 

Those making appraisals, have a responsibility to provide 
systematic information about/the results of p6licieS; and ! 
programs and about .the degree of conf idence\^tt:ached to. thesie 
re'su'lts. Where, a high degree of uncertainty exists , 
preclude firm recommendations concerning policy and program- 
actions. When recommendations vare made in these circumstances 
the uncertainty must be clearly communicated.- ^ Further y. 
appraisals can frequently reduce, the uncertainties and provide 
/a basis for .firm recommendations. \ 



Checking compl eteness pf the appraisal 



It is helpful in preparing an interpretive summary of a 
policy or program appraisal to. view the interdependent con- 
cepts which have been discussed as a checklist. 

Some of the questions which should be contained in sue: 
a checklist are: . .. 

—Were .the reasons for the study found to be valid' 
Were the cause, scope r and intensity of the origin-^ 
problem or issue, redefined as part of the study cr -^^^ 
a result of the. study? ' Why did it need attention at 

^ this time? Was full consideration given to the express 
sed needs. of all = potent ial users of the study? 

:--Were the objectives of the program or policy clearly- 
identified? Did they shift over time? Were there 
implicit objectives? 

i ... 
—Were any special problems, either conceptual, or prac- 
tical , -encountered in using input, process, output, 
efficiency,' or effectiyeness 'measure^? V. Were valid 
standards for compar isons us:ed?_ Was : it necessary to 
employ surrogate measures and what Vas the rationale . 
for their choice? What other quantifiable or in- 
tangible, consequences were measured and- how? 

— Were data collection instr umentis^ sufficient under the 
circumstances? ' y < 

--Are findings statistically significant aind . practically 
■ . important? Do they -answer questions posed at the be- / 
ginning of the study? 

-Were the hypotheses accepted? Were uncertainties^ re- 
sulting, from problems with data ident if ied and properly 
'considered?^ Compared to cOther studies or evidence, do 
data and conclusions agree? If not^ wHy not? 

-Were the lessons learned identified? Can ' suggestions 
be made for immediate improvements? . , ^. ^ 

-To what extent can the per f ormance of this program be 
qeneralizecj to apply to; other settings within which ■ . 
the program takes place or may take place?. What, should 
and should .not be. done in, the future in other locations 
br in similar programs? Are these conclusions ■ baqed 

' ' ^ . ■■" '••29/ ■ : " 
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on demonstrated causal relationships? Are reasons 
for program wea^Knesses incJicated? 

— Have , r ecommendiadrrions been devW^lo'ped • ^or -lalter h^tive.S 
to :6e analyzed ccmparjsd?;^ /--; ' 

— ^l>*tat is ^till left ro be sttudied? Wferat nf^^w questions 
^^^e raised that reqisire further rasearctoi' Which areas 
Si^ reseTaiTch still need further exploration? What 
T4it:t2B3arch methods meed to be developed or improved [tin 
OTuder to make f ut-ryre appraisals more autibor itative? 

ASSESSXJJG POLICY AND PI^ -^ ^RAM ALTERNATIVES ■ 

As in the case of ^.^psraisisisg policy ana pr^r^gr am results,, 
the me±±iod s uSed in ass^*^ srmg. pal icy and prx>grsm xilternat ives. 
build CTi the fundamenta discussed at the beg±iaiming of this, 
chapter. In this cas,e dj^so, there are additional concepts 
and metiiods, which are needed , such as 

' — developing-.a range of alternatives, 

. — screening the preliminary alternatives, 

:~estimating the measurabLe consequences,. 

--^assessing 'provisional orderi%gs, 

/.--determining the -impact of cpristraihts, 

— ^reassessing the ordering of alternatives, and 

— checking the completeness of the assessment. . \ 

. - ■ ■ . _ • ■ ■ ' ^ ,. ' ■ 4 . • ' 

Developing a range of alternatives \ 

It is essential to search out a wide^.range of altetha- ' 
tives. The initial search for alternat!ives should riot be | 
constrained. Continuing , modifying , expanding, reducing, pr 
bandoning an existing program should be included, as vrell|as^ 
ompletely new alj:ernatives . With regard to the existing^ V' ' 
^rogram, consideration should be given to reexamining the | - 
\)^.alidity of the existing objectives. The process of deyelog- 
iWg alternatives should include a: thorough questioning of the 
n^ed for any governmerital intervention ,:whichvmay have been* V 
jlistified on any of the . following . grounds : "7 ~ ' " V 

• T .'■ ' ' . - . • • ■. '• - ' • • . ' • " \ 

1 '—Absence of suitable private alternatives or absence ^ 




of a private marketplace in whicii, the needed service . 
csn be distributed. 

' — The.&enef i}:s r society resultimj froffi ^2a:L'^;^a;;_Jase 
. of services at f acilities , suclh a:s se^wa^ S2i3pDS«l . 

•^-Equal availabil ty of a service, such as .pxEGilic educa- 
t ion . 

—Distribution of benefits to disadvantaged people, snch 
as health benefits through mefi-^care and medicaicL- 

— Esgalation of private activities, Such as the certifi- 
cation of effectiveness and pui^ity of drugs. 

• • • . * ■ ' ■ . ■ 

—Provision of iirc:entives for desired priv.iffite . activities 
siich as develOFTTient of energy resobrces.. 

Broad classes of £approaches which show poteEtial Sor 
salving the problem being analyzed s:hould be initially iden- 
Sfled' One^r more promising alterr^ative . appx^ches.^om, 
aach of the broad classes should, be developed.. If broad , . • 
Slasses are not examined, alternatLve approaches are uaaally • 
fnnlllssar ily: limited to relatively -^-^^ ^'^^""^ 
-from existing programs. For, example , analysis of an incremen 
tat change in eligibility standards for the food s tamp jro - 
g?am is Lre narrowly, defined than an analysis of overall 
income' security or nutrition policy. ,. 

' Reasonable alternatives from all sources, including Jhose; 
suggested by governmental agencies, legislative committees, 
advocacy or^nterest groups should ^considered Issue 
papers-, - such as -descr in chapter 5, can be useful at this 

stage of an ■ assessment . 

ScreeniTig the preliminary a iternatives . 

• A jpreliminary analysis of the likely con^^^^ 
ciated^ith the range of alternatives, including the. status, 
^Suo should now be Sndertkkea. This . initial screening is - ^ 
intended to eliminate obviously infer ior approaches .^nd to- 
reducf4hroriginal list of alterna-aves^^^-t^ manageable size, 
K is helpful to make approximate cal-culations^of cost.and_ 
consequences,, of break-even points, .-^^ 

i tv etc "Alternatives should not . initially be rulgd _ out 
i^s^d^n' implementation difficulties, including organizational 
or procedural changes. . • . 



Modifications , and : ^inations of alternatives usually 
become apparent and fr^q r^ritly provide the basis for new and 
superior alter natives i i^e^ searcia for, alternatives iiis d 
continuing activity and :rx^-r analytical effort provider 
opportunities to invent r discover other alternatives that 
will arise . 

Estimating measurable cc-ise^uenceB 

Estimates must be m-^vd^: of anticipated measurable conse- 
quences as well /as of ail ^i^ssts and resource inputs under 
various conditions and "l^ f-^is of available resources. MBasur- 
able consequences includes *i2£f,ectiveness , side ef f ects.r and 
d istr ibut ion cons idera t£OT\^ In making such estimafcsis , the 
data on actual costs anfi errrec t iveries s found in- prioji: apprais- 
als of similar programs sssxoild be used together with actual 
opera:ting data. It may ^is^:- be necessary to use well developed 
..causal models to make sucm effectiveness estimates. Although 
these models„ must adequatteO y simuLatte the real situation, an 
existing modei may server- Expeniemce has shown . that- it is* . 
costly and time consuming to jjaeveicip a completely new model. 

Some effort should be made to ^estimate side effects and 
their influence on resources. An est imate is needed r to the 
extent possible/ of the differences of impact on the benefi- 
ciaries- and the cost bearers (distribution considerations). 
Approximations may have to be used ±or side ef f ects ^^ad . 
distribution estimates, and various value judgments seb:*" 
involved TrT -weighing both. 

When analyzing ,costs which- should be associated with 
.effectiveness r var ious cost concepts are needed/ and informa- ... 
tion on these costs, is usually available. When analj^zing 
costs^ which should be associated with side-effect and distri- 
bution consider ations r tp'tal as well as incremental .costs 
should, be'/developed . .- Frequently such costs are incomplete. 
They should be checked for . reasonableness and consistency with 
the alternatives of interest . 

Information at the margin.^ contrasted with information 
on total quantitiesr is very iprportant in resource allocation 

decisions. . Approximations of incremental costs , however , v 

are more .easily obtained thaxi are approximations of the mar- * 
ginai aspects . of/ other proga:?sm ccmsequences . A; reasonable 
effort shcSuld be made to eslitianate the d irection and magnitude 
of the variations, of Tprogram jCDnssquences over relevant ranges. 



Information/oh measurable consequences obtained :from 



audits, evaiations,. or other studies should oe used. Histor- 
ical d. trend data may provide ^nfprmation concerning how the 
varic' . consGcuJe-nces.. are affectec by the, scale of acti^rity. 

Assessi ng provisional orderings ■< 

■ Once t2e ^tal and incrensa'^al consequences of the 
alternatives have been estuna-xei, the alternasriv.es . should be 
airrai-cJ. in SQ=me order . " This oxi^.,: ing may be oased on one of 
sever c.i-'avail-a;rle approaches. • ^ 

One a^EXOoCh is "cost effectiveness. "••This approach 
focuses on res.Gac-ces expected, to De consumed ana how well 
the objectbxes are achieved. Vs±nq this framework, a P^e- 
ferred alter-native is . identified as one which produces^ the _ 
largest achievement for a given level of costs or whxnh mini 
mlzes resoux-ces expended for attaining a given level of 
effectiveness. 

' While ^he cos t-ef f ec tivenesrs approach provides a basis 
. for. ordering competing alternatives, it does not ^j.^^'^^y . 
allow for comparisons of alternatives.associated with multiple, 
possibly conflicting, objectx^. Other consequences of 
alternatives— side-effects ana distribution c^onsiderations- 
are not. an integral part of the anaalysis , and may require 
separate ^examination.. 

A seconnd approach to ord:er:rng: alternatives is "cost- 
benefit" analysis. ■ Side-ef £e:crs:. snd distr ibutxon considera- 
tions are iincorporated in thi^: approach. Major consequences,, 
prSenefitiVarS measured in dollars, and differences between 
Monetary benefits arid costs i^iwide ^he basis to^ choice ^ 
among alternatives. • Theoretically cost-benefit analysis 
is more useful than cost-effectiveness analysis in treating 
differing as well as convicting abjectxves.. The .streams of ^ 
benefits' and. costs, can be diac:o:onted to their '^equiv^ 
ent values, thus accounting for the effects of time. ConCep 
•tually Scisionmakers could sel.ct progxams based, on rankings 
of net preset value benefits (or derivatives of this data) 
Sntil the ?^al available resaxircfis were e^-aus ted . However, 
?his approadh. requires that all assures be converted to 

dollars (a (^fftcult task at best) .m a w.a^- which the deci- . 
lionmaker understands. Because, of the d^culty of quanti-^ 
±Yimg side-effects, and distributional effects, there is often 
no clear di^siinction between cost-effectxvsness and cost- ■ 
beTTfifit .anaH^'sis . 

&ji6ther -approach^ is '"rcost-vaiu'e" a^lysis . This is a . 
• teciaiiique Smr obtaining gsneii^y acceptable quantitative 



w^ighrt^ for use in comparing the value of the .alternatives. 
1:1 t this approachr the weights assigned to various outcomes, 
are |v*ased on decisiomnakers ' judgments. 

The co'st-'value method combines elements of cost— effec- 
tiveness and cost-benefit analysis. Side--3ffec± and distxi^ 
tixitia)?;! considerations can be incorporated with effectiveness; 
Becaitifi^ti the judgments of decisionmakers differ ^ various sets 
of jiiS^.'^igments should be used and the ordering(sj of aiterna- 
txx?^^ should be tested for its/their sensitivity to these 
dafi55?M ences . In such analysis both'the array of consequences 
assa^iated with each alternative and the ordering based on 
tteer:^rious value systems should be presented to decision- 
'maJce23 3^.. ... 

2ach approach has both;i^strengths and 1 imil:at ions., but 
all siiare certain limitations. One such limitation is^ uncer- 
caused, by such things as variations in assumptions and 
the imaLity of information on the alternatives . Becssise • 
uncHiita^iinty is always present in antic igatrng f utu re zouut comes , 
undu^xiel iance should not be placed on smalX ciiffereircB^ «in- 
ordexing(s) of alte5::nativas« The quantitative .analysdzs which 
■has iaBien d iscussed should -be supplemented wzuth ran aniSLlysis of 
nonm^surable consequences. A serious attempt i^hould .be made 
to ind icate the':signif icance of nonmeasuraMie sronseqa^ences . ' 

Determiining the impact of .constraints • "/-f 

Special efforts should be made to assesi's rth-e. imgact of 
actuial and potential legale financial , and poiZttical icon- . 
stradntis. Programs and policies must operate mthin th^e 
frame^prk of* law.. Alternatives which may. app^r theo:ret ically 
desdrable must also operate within the law. CSujnsequently f 
tis: alt\er natives considered for adoption must conform to this 
feamewojrk. . ' .. * - 

In addition to th^ls^ sorts of constraints^ ther-e are 
csims±xaints resulting :&:<im conflict with otteer objectives, 
to: e:xample of such constxjaints is the conflict bsetwaen errvi- 
rxanmentalf transpor tatdouy and energy objective:. 

However, constraints are not inflexible. T::f dsoisd^n- 
maksrs were clearly aware of ^ the potential apja^rtraritiies fore-r 
gons; ;r esoxlt ing[ from existing constraints, tfeoise constxiaints 
migfiEX change . ' '■ . 

Bec'isionmaker s must consider, possible public reactioh 
•to sUter native policy ?and program options, strategies: that , 



migh^.. increase their acceptability, and ^^a^ administrative 
or Viier operational ba-rrierS:to implementation exist. The 
proSems ofamplementation and of ^^eptability may, to some 
degr^^, be dealt with in analysis. Usually, assistance can. 
be pSvided to decisiomnaKers in identifying the "second or 
thirSbest" alternatives which may have higher prospeats for^ 
being accepted or implemented. vv. •. 

It ii-as -been argued that if acceptability considerations 
are avatded', the assessment of alternatives becomes more 
Sb>rtlTe, less.paarochial, and less tailored to fit precon~„ 

■ feivll p^sitioas. on the other hand, it .:.ay ^^^'^^U^^^^^?^,, 3 
If acce^xtabilitry consider actions are not included, the analysis 
55ay prove to be^irrelevant. 

Reas^ssinq the order i no s of tbie alternatives. > 

Srderings of alter naitives are always provisional. They 
are .dSe?mL within the context of the '-^^^ y^^^, 
considered to -be important during the co.urse of the analysis. 
Ihr^umptions- and values underlying the various^orde rings , 

■ of tS alternatives must be cleaxly presented to decision- . , 
^fkS. Fuxth^more, even when the a^ai^st thinks, the study 

caampleted, c^cisiomiraiteers may .raise new_ic-ues, asK new 
^ueSInsv renniKst farther .study , and ask -for additional , 
?o^±so.ns. thes^ requests are answered , the . orderings. 

of aOater natives may shift. • 

Although attempts shoulii be naie to include as many .^^ 
fact«3irs as possible^o^ex considerations properly affect the 
final policrana pragram choices, .- Some of these considera- 
tions may |LpLtel:5-:t.e^^^^ iTonfln the 
ability to estimate, ex/en fflaalitatively, and^belong the 
S?ovince of the decisionmakers' judgments. .However, the 
SSst should ^tempt to .miderstan .. these ^°^=t^^if - 

to devise se^^tivity anralyse.s v^lcb inay,::help to^sharpen 
SS decisionmaker- 3^^m^t:s - rm^^^ 

5^- altennatives «tiicb -iDaSsnce ^^'h?-^^^^^"* . ^^^^^ ■. 
atrrectivss in wasrs-nofc 5)eir.cCTed when initial set ot ^^ 

^SStives was~devetepBa: with the decls lonmakers ,.: or they . . 
miv^entify alternativ:es which keep options open or avoid , 
T?^everSb!r damage or risk. . Experienced ^^^^""^^^l"^ 
dQm-a:ttempt to use an '"^plrimizing" technique-, for this sort 
o&™umicatio^n,. with dB^isionmakers because^many of the 

■ impoxtant considerations are neither specif ic enpugh nop 
q^ntif iable . - Both Van^ysts and , dec^ionmaker? must._ be ^. _ 

. satisfied with what, irr. their 3 udgmenit,, . is^ a 'good but not . 
necessarily the "theoretically be^^s^^^^^ . 
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Checking cOinpl et:eness of the assessment 



' ■ • • * ''' ' 

'Sortie of the questions which should be considered in 

preparing an inter pnetive- summary of a policy or program 

assessment are contained in the following checklist: 

— Were the reasons Edx the study found to be valid? 
Were the cause ^ scope.^ and intensity of ^the original 
probilesn or issue redeEined as. part of the. study or as 
a resnxt of the study? Why did-it need attention at., 
■this time? Was full consideration given to the ex- . 
pressed needs of all potential user s of the study? 

— -Were ciie objectives of the program or policy explicitly 
stated and validated? Did they change during the 
course of the ass:€BSSTnent? If so^ why? 

— Were tihere any potentially interesting alternatives 
eliminated early in the analysis? If so^ why? Under 
what circumstances might they become attractive? . ... 

— -Were amy special prtsblems, ^eith^r conceptual or^iprac- • 
tical / encountered in specifying, an adequate set of., 
quanti±iaJbiLe measures? How reasonable were the dollar 
values attribatHd tin physical measures, if .that was 
doTi;e? Were gnalita:tive indicators proper ly -identified 
and used? ... • . ^ 

— Dtj the effcec-td^'E^sne^ :ii^asures accurately reflect the A'T. 
degree attaiinitesrt cuf the objectives? Wer e ^they ■ :tl^ 
c2I^^:sis:^ i^ntly a^d; ^oszg all of the alternatives'?^- ;is. . 
tiEE ei:iectxv^nes^r::^tSL reliable?^ Has uncertainty in 
t±ie diata been prxgEsrly considered? ; , " 

— W^]^ side— ef f exrfcs ^nd distribution considerations ade- 
- guad^Zlj considi^r^J Are there significant differences 
o S3233:n5 t!xe alteiTEatLues? - " 

— Were aim of the aoi^t impl icatibns captured? How reli- 
able a-r.^ they? Wb^^t is the range of uncertainty? 

— To\iaa:t assuinpti.oi23 or data is -the ranking of the ^ 
alt^rqiatives serisitive? Are there any actions which 
•csnci mate the iesading. alternatives significantly less 
a£fa3:Ss&d by^ ^e amcertainties? ^ 

= -~A^iSr ^^ttltei:^ ^any sr^^cz^l problem.-s connected with' gaining. 



geneL-al acceptance of the apparently preferred alter- 
natives? Will implementation of &ny of these pose 
particular difficulties? 



■Is it likely that additional information about the 
leading alternatives would change the ranking? How, 
when, and at what cost could this ihformation^be ob- 
tained? Can the policy or program decisions be held 
open while-new studies, evaluations, or rese.arch -ef- 
forts are completed? -what lonST^erm evaluat^Lon or 
research efforts need to Tbe initiated to me,^$,::Jj:i^^ 
or related problems in the future? -'u,. 



CHAPTER 5 

PRACTICAL ASPECTS OF MANAGING AND- PERFORMING S TUDIES 

Previous chapters have discussed the conceptual aspects ^ 
of appraising the results of policies and programs and of 
assessing alternative solutions to the issues related ,to 
them. " In deciding' which problems to. study and Tn carrying 
out actual evaluations or analyses, however, certain practical 
questions arise which need to be addressed. This chapter 
discusses some of these practica.1 aspects, and offers sugges-- 
tions for coping^-, with them. The list is not extensive but 
is indicative of ^ the very real problems faced in this type 
of wor k . 

FORMULATING AN AGENDA OF ST'UDIES / 

One of the most important responsibilities facing ^any 
manager of an evaluation , analysis^ audi^t, or other prog'r.am . 
review staff is^developing an overaLJ. work plan for the organ- 
ization. Plannilfig a E)rogram of studies which will be of maxi- 
mum benefit to decisionmakers .should involve two principal • 
tasks : 

— Identifying ..problems or issues which ar.e evolving as 
major areas of concern. 

— Deciding Which of the many cand idate , problems the 
organization should commit itself to studying .V , 

' ■ ■ ■ ■ ■ 

Identifying emerging: problems 

.A contribution can be made to resource allocation dec i- 
sions by rais ing problems and explor ing the it ramifications 
in issue paper s. " These focus bn problems which, there is 
reason to believe, wiil become the subject of a full-scale 

evaluation or analysis;. I 

j ■■<■.■ 

The ability to recognize emerging 'problems for issue 
papers depends on experience and^^^good .judgment . Developiing 
."an issue paper .also requires an understanding of the problem , 
area and its environment. 

: An i^jsue pap^r may follow the format and style appropriate 
to. a f ull-^s"Cale -evaluation^ o analysis but is* limited to an 
assessmey;»t of what is known about the problem. An issue paper ; 
could-be as short as ' a feW paragraphs .or \l.ohg enough to cover 
all:;or almost all of the points required^in a full evaluation; 
or analysis but without the sc6J>e . or definitiveness of a 



finished study. An issue paper should emphasize r-.^ commenda- 
tions on the Lture of further study efforts. Por'-example, 
whether the problem should" receive high priority attention (and 
why), whether it should be pursued but on a long-term basis 
(and. why), or whether it should be abandoned (and why)... . 

D eciding which problems to study ... 

Many problems, programs , and jpol icy. issues are need ■ 
of systematic study. , Yet,, scarce staf f \resour ces need- to be 
allocated to the most productive projects. 

Issue papers can identify policy and\program problems 
worth evaluating or analyzing. However a complex series of, , 

•judgments is still needed to select' that particular group of 
problems which, if solved , would : maximize the ^anticipated . 
payoff. While- It is • relatively easy to list- the factors in- 
fluencing these -choices, it is seldom feasiible to appraise 
rH o? ?h-em in a formal quantitative fash.i/on. In^some cases .... 
there can be little more .than an informed^guess about- the 

. potential, utility of a fetudy. . / ,. ' , 

; A systematic- weighing of the .following .factors win be 
helpful./ .." - 

-.' .1. The anticipated .payoff of successful evaluation or / 
analysis . 

- ^ This/payoff can* take sever al -forms : an ineffective, pro- «• 
gram can be canceled ■ and costs saved ; ^ "'^J"^^"^^^^ P^°?^^"^_s 
• can be reshaped with consequent improvements in effeqtiveness., 
reductions in costs, or both; or better alter natives . can be 
substituted for cur rent programs and policies vrith gains in • 
effectiveness, reductions in cost, or both.. 

2. The chance of the successful performance of Snreyalua- 
tion or analysis. ' . . ^ „ V 

^ This judgment d'^pends on a basic understanding of the fun- 
damentai causaT relationships; the requirements for additional 

^ Information; the adequacy of current f^^^^^^^ J^'STliS^ - 
Equality of staf f consul tants, or contractor s ; and the . time ^ 

and money available. ^ 
.:- . ' 3... The chance that a -preferred course of action can 
actually be implemented. . -• - , - 

■ .^ -. ■■. . ^ . , ■ .- ■ 

. , .,32 , •■. - -. 
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This judgment depends upon such things as newness, 
simplicityr visibility, coverage "and. timeliness of^ the/ ' 
preferred course of acti^on* . 

4. The need for resolving the problem' or issue.. 

This need depends -on. the nature and relative x^mportance 
of the problem' and the time remaining before a meaningful : 
decision has to be made. " . . , \ 

■•- •■ ' . ■ ■ ■ ■■ ■ \ ■ ••. . . ".\ ■ 

'5* The estima.ted cost of the evaluation or analysis. ; .* 

. •'■ " ' ■ ■ - - '■ ■ ■ . ^ ■ - . -.^^-'v ' • ■■ ■ : 

BEGINNING A .STUDY . > ' " ^ / . , " 

Certain tasks need to precede major commitments of staff 
and other resources. These tasks include preparing a study 
plan, obtaining necessary agreements, selecting the ; study 
team, establishing lines of communication , and selecting ^ 
appropriate methods^. ^ . . : 

Preparing v-a detailed study plan. , . ^ 

Substantial effort should be .devoted to drawing up a 
comprehensive and thorough s.tudy^ plan which will serve as a ^ . 
guide for all' subsequent work; A study plan, that is too k 
broad in scope or loosely stated is almost certa^injtp create 
false expectations for somje interested groups. . Clearly , trade-- 
ofts have -to be made between the ^time devoted to planning 
versus doing a' study and, within'the planning period, between . 
a detailed and a. general study plan. . ' ■ : 

As the study pragresses, it >is likely to deviate from : ' 
original expectations. ' teer.haps , the vissiie turns out to' be 
different from that or iginally postulated ; the^ objectives may 
not have been stated precisely enough; a working assumption may 
not prove viable; other alternatives to the/ program emerge; new 
facts come to light; hoped "for data cannot be obtained; and , so 
^.on. All o*f these 'developments call for some modification of- ' 
the study plan. Changes should be made, aS appropriate, to the' 
•study plan. - , ^/ 

Es'sential el'^ments of the study plan would appear to be: . 

■ — A clear statement of the problem to /be s tud ied , 

questions to be answered , and. decisions to be ^affedted. 



--A careful listing of constraints and assumptions.- 
^ — A sta^tement of methods to be usedi. 



—A" specification of the resources to be committed (in- 
cluding identification of the key staff members and. 
any contracted tasks required ) . ' 

~The^f*r'eqaJn^^, f ormat'/' ahd, recipients of reports.. 



— Procedures for amending the study plan. 

— The timeframe for the major components of the. study and 
.the final deadline .• " 

When/ a study or a major part of it is to .be performed by 
contract, there should- be discussion and "understanding by the 
parties concerning the essential elements of the; -study plan . ■ 
This is likely to require lengthy dialogue with- the decisxon- 
Ihakers. Per sp'n& with official responsibility for. the. policy 
or prog ram ''Vnd for the study should ass„ess feasibility and 
validity of the study plan. .Any differences' should be resolved 
before the study begins. Substantial time and effort may be 
necessary to arrive at a workable understanding. In the case 
of contract studies, the agency staff must be technically 
compe^tent to oversee the study and must also be famil.iar with 
the various rules on' contract management. 

S electing the study team • 

. ' Most analyses or evaluations require contributions from 
several key persons. For large studies, subteams for parti- 
cular aspects may be required. As in any group effort, some- . 
'■one-must be in charge to (1) provide guidance, (2) manage .the 
work on a day-to-day basis, (3) report , to" higher authority, 
and (4) generally be responsible for .meeting the terms, of the 
study plan . 

The coordinator or director -should be experienced, with 
a technically sound'' but broad background , [an instinct for ; 
the principal issues, andjthe leadership abilities that elicit 
from the team members their 'best efforts. It usually turns 
out th-at the "team coordinator or director will have to be- 
principal editor of the final report—so wri:ting skill is,, 
necessary./ " 

A-^^eam studying any complex policy or program should 
be c-ompoised of experienced persons from various disc.iplines, , 
with the' stature required to obtain needed information and 
assure credibility of the study. Regardless of their origin, 
however ,f3ll should be made to feel as coequal members of an 
excitinq intellectual experience and useful endeavor. 
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One way to create such an environment—at the same time 
avoiding duplication of effort— -is to have an initial briefing 
on the terms of the study plan with all team member s Impor- 
tant aspects such as concepts, assignments, schedules, basic ' 
assumptions, need for personal and agency coordination , and . 
reporting requirements, should be fully understood .and agreed 
upon in advance Provisipn should be made for per iodic br ief- 
ings by each specialist to the team as a whole so that everyone 
has both- a grasp of overall progress and a chance to offer . , 
facts or insights on any aspect of the study. 

.. It is often helpful to obtain reviews by competent and 
widely, recognized independent prof essibnail analysts and 
evaluator s and exper ienced program administrators . This 
advice adds i;. seasoned viewpoint which may improve the . tech-. . 
nical aspects and may assist the supervisor in assessing the. 
technical adequacy of the work of sfeaff members trained in" 
dif f erent discipl ines . ' ^ 

-■'•/■'•' . ■ . ■■ ' . . , ■ . ■ 

Establishing, lines of communication , • ^ 

If the study effort is sufficiently large, official point^ 
of contact among various intereisted groups and users of' the> 
'study should be designated., This should insure that cpmmuni- 
cation^ of all kinds flow quickly anc^ clearly among tl:\e groups 
having a major' interest in the progress of the study. Open 
communications provide the basiSaf or a more complete assessment 
pr appraisal and a climate in which recommended ghanges are 
mOi:e likely to be accepted and implemented., 

, ■ / . 0. : ' ■ ' . 

Selecting appropr^iate . methods 

Analytical methods which yield valid and (hopefully) 
unequivocal results should be used. However, the method must 
also satisfy the constraints of time , money, and data peculiar 
to the study. If the constraints imposed are so rigid that; 
the stqdy would be compelle'd to use" methods judged to be , 
analytically inappropriate, the study should be undertaken 
only after fully informing responsible authorities of the risk 
that^' reliable conclusions and recommendations are not ],ikely. 

No particular approach or technique is inherently the 
appropriate one. In practice, there are too many attempts 
to moid the policy or program issue, to fit a specific tech- 
nique. This should be avoided. For a specific study, various 
approaches, each having its own particular logic should be 
considered. Usually/ a blend of methods and. techniques will 
be required to. provi,de insights into the full consecjuenses 
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of the various alternatives. Reasons for selecting a partic- 
ular, approach or blend of approaches should be clearly, stated 
so others' can understand the rationale for the particular . 
choice . V • \ , ■ ' 

Whatever apprpaches and methods are selected ^ 'they should 
satisfy' the following criteria: 

" . 1. Validity — how much confidence is th^re that the 
results can be actually used? 

.2. Relevance — aVe the results useful to decision-, 
'-.makers? will the method be used (such as a model) 
to answer decisionmakers' questions? 

3. . Signif icance--will the results go beyond what is 
■ apparent from direct observation? Will the* 
results/tell the decisionmaker something new and ^ 
important? ;''- ■ ,^ - ■ • 

.4. Efficiency — does the value of" the ins ights .exceed- • 
the "cost of- using the approach? 

5. Timel^iness — will the analytical information . be 
• available in time to meet a .management or legis-r 
' . la tive decision point/ such as renewal of expir ing 
. legislation? > " ' ./ 

-Modeling and statistical inference are two related methods 
which are. particularly useful and. are frequently used by 
eval^^tors and analysts. . , ' ' 



A model^is an' abstraction .from or a representational • 
the key elements in some • real^ . system . If the key el^^n^nts and 
their relationships are adequately spec if ied , rel^-ant, and: 
valid,' a model can predict the consequences -of^u-rftried alterr- 
natives and variations in data and assumptj^ivs . - , 

Statistical inference techniques are* widely used to 
analyze data obtained' from .the var mu^ collection instruments 
and analytical models. However , ^6nd it ions .and assumptions ^ 
underlying these methods must satisf ied . if ' the method is . 
to be used. Mistakes can oce1j r , for example, if .prepackaged 
computer programs are used^ithout understanding the assump- 
tions and conditions. 



CONDUCTING A STUDY , . 

: 'In the performance of any evaluation or analysis ^ 
practical decisions' of many types must be made^ and practical 
problems are frequently encountered. Some of the most common 
ones are discussed here. 

C ollecting relevant data . • 

In performing studies^ there is often a temptation to 
collect all of the information which might ;be of use. While 
every p.iece of information . may have some value in tiie right 
placeV' is it relevant and worth^what it costs to acquire it? - 
Quisstions which should be cdiitinually applied to any data 
collection effo^r^'t are: . ' 

— E^ar6t"ly what question is this piece. - qf .data intended 
:o answer? ^ 

— What analytic model demands it? 

— What calculation cannot be done without it? 

Testing the reliability of data 

^ An attempt should be made to estimate v^hether data is 
reasonable at. the time it is first generated; i.,e.,;^how does 
this new piece of data square with everything else that is 
known or can be deduced relating to it? This is especially 
important when complex calculations ' are involved. How doe s^ 
the answer compare with rough ^calculations? The exerc^ise, of 
making rough calculations frequently gives, the staff member, 
new insights into the data--* \ ■ 

There are ^numbers/bf one kind or another which are w^idely 
published. everyone • seems to use them unquestioningly.\**^H^^ 
ever/ a car eful analysis has often demons.tr ated that some data 
has a di^fferent interpretation than what is commonly supposed. 

Occasionally f an attempt may be made to withhold infor- 
mation./ It is not uncommon to hear that data 

(■ y . . ■■ . ' ■ • ■ ■ 

y^^^ — is too hard to assemble r 
y" • ' . ■ .. \ . ; • . ■ * 

—does not exist in the form wanted, . • . 

— is only a working paper^ or 



— is privileged . 
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When faced with this type of situation, the analyst , 
should (1) consider the value of the Informatipn to the study, 
(2) attempt to obtain a release of the appropriate information 
if needed, and ( 3) propose to the study coord inator that a: 
•formal request be sent for the ..needed . information . In some 
cases, essential data will have to be "constructed or 
"extracted" from "secondary sources.. 

Frequently, data collected from different sources about 
the same subject matter will be in apparent conflict. The 
first practical step in getting the 'r ight data is to recon- . 
ci-le the apparent conflicting interpretations of the^data. 
An appropriate question may be:.., Are they truly, two different , 
sets of values describing exactly the same , event or situation? 
A second step would be to examine how the data.was^der lyed. 
The apparent: conflict may be a simple funcrtion of the^data 
collection methodology. After these procedures have been 
employed, it may be appropriate to use an analytical technique 
to determine the significance of the .differences. Additional 
assurances may be obtained by having _data reviewed by experts 
in the field.- ' ' ^ 

Protecting the confidentiality of 
Information about individu als^ ' 

It is often necessary in evaluation and analysis ^to 
collect data about individuals. In most research involving _ 

■ human subjects there has been a firm commitment to protect the 
confidentiality of personal data. It .is important to make ^ 
certain that data on individuals is not personally identifiable 
in the study- or in. unsecure files. If-it is necessary, to^ - 
obtain information from the same individuals in^subsequent , 
time periods, special controls and procedures should be . 
required to assure that systems of records do not. disclose 

'-individually identifiable data. 

Federal agencies and some Federal contractors are require- 
ed to comply, where applicable, with all provisions of the .. 
Privacy Act of 1974 tb protect the confidentiality of individ- 
• ually identifiable data. „These provisions include • 

—public disclosure of the fact -that ah agency maintains • 
a system of records about individuals, 

— strictly enforceable procedures for assuring that 

individuals have access to their records and the oppor- 
tunity to correct them, ' .. • 
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controls on disc^losure of individuals' identifTabT^ 
. data, and ■ " *. 

— administrative r technical;, and physical safeguards 
to p^revent unauthorized access to suet) data. 

In planning a study, care should be taken to require 
individual identifiable information to. be collecl^ed oply 
when no -other approach can enable the issue to be validly. 
studied. When such d^ta is collected, it must bie. properly 
protected . • \ — , 

Documenting and referencing . ' 

. 'Documentingy appraisals of results and assessments of ' 
alternatives is important. The; dogumentation; should be 
sufficient so that another individual or team involved in 
reviewing .the' pol icy or program, by reviewing the documenta- 
tion, could follow this analysis and , as needed , reconstruct 
parts of it or use it in another study. Basic assumptions 
should -be clear l-y identified and recorded-. The rationale 
for using, direct or surrogate measures should be stated- - 
explicitly. Qral interviews shoUld be summar ized in writing, 
dated, aind filed. Original docuii^nts should be reta ined. 
Complete files of re^levaht taw tialta and work papi^rs should 
be kept and filed so that fciiey cam be retrieved easily for 
re.view. ' Information which cani^t be; readily f iled should 
be adequately described and referenced in the files. 

The study team should c^SHgn,\ use, arid save work .paper s . 
Well, designed, clearly labe3teS;,. and fully, legible work papers 
•^Q'ffer an important insurance policyVto., the study^^ t^ The 
work"* papers constitute the evidenqeAga ther ed . A review of 
the work papers wi^l show whether the study team has been - . 
thorough or whether they may have ove^rlooked an important fact 
or element . of Va^ problem and that all similar elements of . the 
analysis or evaluation have been treat:ed consistently . The 
work papers should be checked aga ins t the . study^ plan to assure 
that the plan was carried out or that changes are fully ex- 
plained . Developing the total costs of each o£ a ser ies of -"^ 
alternatives is an outstanding example of the need for , and 
usefulness of , a caref ully designed ^and clear ly labeled set 
of worksheets. ■ Without them, the chances,, of missing an' impor- 
tant cost element, incorrectly calculating an intermediate 
result, .or costing, the competing alternatives inconsistently 
are substantial. 

Work, papers should be dated and signed so that a clear 
trail, is established as to who did what and when. The best 



way to tie it all together is to file;«ith workpapers, one 
copy. of the final report which is cross referenced to signiti- 
cant sections of . the workpapers. ■ . ; 

Adherin q t o time 'schedul es . '.. 

• Effort should be made to anticipate some of "the possible 
delays-, and the time schedule- should allow for unforeseen _ 
delays. 'Most complex tasks are harder than, oragxnally anti- 
cipated and, therefore, take longer than estimated, In complex 
studies, detailed schedules for ' component parts may be neces-. 
sary. A proposal to expand the scope.;of the: s.tqdy or to ,do 
more work in order- to sharpen the results sho:ald be carefully 
justif ied,, particularly if, it; involves risk, of delay m the 
schedule . " ■ . 

Leading and coordinating' the study team v. ~" 

.It is essential to maxdmize "the interaction among the 
study team members, Ehysical arrangements which 1-"^^^^*^ 
this shoold be avoided onmodified If at all possible When 
gathering the first list of alternatia^es or hyp^otheses, brain, 
storming is extremely useful.. . 

The coordina-tor should take, every practicable step to 
insure easy access to the diecisiorimateers who expect- to use 
the^analysis or evaluation.. A contirming (but not necessarily 
cSnttnuoL) dialogue should, help to :nmke the.products^us|f.ul 
and well accepted. The coordinator .also oeeds to impression 
the team the importance' of maintaining an °P^"r^°"^f|'^^^^ 
amicable relationship with the personnel of the program under 
anal?sil or evaluation., It is all too easy for program people 
to frustrate a study if they have been antagonized or hurt. 

Usin g computer-based models 

' For most large-scale; but roatine, quantitative manipula- 
tions (statistical analysis, linear programing , etc .) good 
"canned" programs are .available and should be used. When^a 
program or problem has many complex interrelationships, how- 
ever , and the effects of altering the assumptions or data are 
not Obvious, a .specially designed, computer-based model may_ 
facilitate the study. In such case,s, creative computer pro 
gramers aire extremely vraluable. 

■ The structure and operation of any model should^be ' rea- 
• sonably .apparent to decisionmakers who want to use, the. study:, 
•its output and workings .must be readily understandable to 
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them. Usually , this can beV^^^ by carefully dia'-/ 

graming the components of the model and explaining how each 
component operates and interacts; -with th^ 

the study will normally accept the computational competence 
of " the \model if the .logic; makes sense; to them - and they .have 
cohfidence in the study team. 

COMMUNICATING STUDY RESULTS . 

Many persons doing stud iesVf ai.i. to understand' that doirigf 
a good piece of work is necessary but hardly sufiEicient, for . 
bringing about a favorable chang.eV^ in the world At least 
two major steps beyond successfuT'compie^ a stud^ are : 

rieguired: the results must be clearly , ;Concis^ly/;^^a 
"g:en.tly communicated to aid those affected and .^;a policy or " •• 
.pr:ogram decision must-b made which results in /some k^^^ 
inaction.;-"''' y ■ 

jSpecif y ing the nature of reports 

•• . ■ y i ■ ■ . . ■ ' ■ . ■ ■ ■■■ . ■ ■ 

' There aire three geheral classes of problems involved in 
^repor ting appraisals of results and assessments of alterna- ^ / 
tives.: /(I) to whom reports should^ be made , (2) when reports 
should be made/ and (3) what, style and content char actet izes 
good . reports . Each new study will suggest its own individual 
requirements and should be made ' a matter of recorcl in the 
agreed work plan, adopted^ before each study :ls begun. A few ' 
general guidel ines can^ however ^ be set down. ' 

Obviously, the final report should Be addressed to those 
who 'are in a, position to take appropr iate action'— or to assure 
thyt it is taken by' dthers. .;vUnless special consider ations 
c3ictate otherwise (e.g. secur ity problems ) , reports should ' 
routinely go first to the- team supervisor^and others ^9 needed 
to insure that they meet the organization' s prqf ess;io.n^^l st^ 
ards . Even prof essionally sound stu;d''ies , lh»owev?.er r %^ result 
in disagreements with the managers of the pro.granis being 
studied. In these cases, the study, team should reduce the 
number of areas of disagreement; and, where these coixtinue to 
.exist , the issues should be substantial and clear-ly Sef ined . 
Although decisionmakers waiting, to use the report stooiiild be • 
kept informed of key bindings, it will in the; end seiH^e them 
best if the review process is complete before the final report 
goes to them. : ' 

No report, other- than the final version,' should be distri- 
buted beyond , those mentioned above without their concur f ence . 
Unauthorized release^ , of preliminary , draft, inter im , or partial 



reports can be harmful because, frequently, erroneous,, infor- ; 
mation, even though^ corrected later; becomes wrdely diffused 
and becomes a source of further exr or anc3 confusion. Publicly 
available reports should be. free of such' errors.. 

• In planning the study , ' suffi,xieTrt time should be allowed • 
for Writing the final draft reporc,, gathering comments, eclit- 
ing, and securing the necessary approvals-. The report writ- , 
ers , in turn^ have an obligation to 'complete t,l?e report within 
the scheduled time. Report outlines should be prepared. early .• 
They can provide indications: of the most critical data gather- 
ing and interpretation tas|s yet to be coinpleted in order to •' 
have a useful and timely repor t ■ Decision points come and go 
relentlessly, an'd a potentially good, decision-af f eCting report 
may rbse much o'f its value because it was not available when 
needed . . 

CommJnicatihg with clarity an d conciseness . ' 

Writing a. good report is an art,, and the required skills 
are probably as scarce as those necessary for evaluation and 
analysis. ^The solution' is' to insist that staff members work 
at learning" to write well^ .One-helpful step is to provide 
staff members with specific guidance,, such as a good style 
manual, and insist that they study and use it as part of 
their regular duties. In addition, someone on the staff can 
serve as resident editor. It is frequently helpful to have 
a skilled technical' editor or writer joiii the team when the^ 
report is being ■ wr itten . All significant alterations should 
be discussed with the author: not, only to insure accuracy, 
but also to assist the author in learning to write shorter 
and more trenchant reports. 

Study r'epotts are typically, directed at a reader lacking 
relevant technical training. Therefore, the main body- of the 
rfoort should .be written, so ttaat it is r eadily , compErehen^ible 
to^the nonprofiessional reader^ However", mater lal , included, in 
the. report . should be sufficient so that a. reader can understand 

• the arguments in support of thie. conclusion. ^ Jargon should.be 
kept to a minimum, and where it is listed, define it carefully . 
Supporting fcechn ical mater ial ■ should be presented in appendixes 
Graphs and tables included' in the main body of -the text must 
be clearly labeled and fully discussed in the text. Shorty 

. reports are typically self-contained, while long .ones. ought to 
be accompanied by an executive summary of the study's general ; 
conclusions and 'recommendations . , 

' ' There will, of course ,■ be d if ferences,;'between the format 
and content of a report on appraising program or policy results 



and. a report: on assessing aLternatives , or a repor\ containing 
booth. Within each of thes^e, some variation in formal and con- 
tent is inevitable r depeitding on 'the nature of the paiicy or 
program: issue being s±udiied anc3 the methods used. In^eneral^ 
the foxjicat and content Oif reports should coyer -what wa^found 
through- each of the concerts and methods d iscuss.ed in chapter 
5f as- appropr iate . , * \ 

Foil owing up 

" Writing a clear, coneisey and informative " final" report 
is not the end of the '.Vcommunicating" responsibilities. 
Usually, some decisionmak^s will, need assistance in (1) inter- 
preting the report / ( 2 ) Jilar if y ing aspects of- it, (3) getting 
answers to questions ran^Bd by it but not answerec3, and* (4) in . 
general, developing a rssoned reaction to it. Briefings, > 
informal guestion and airs^r sessions , and various kinds of . 
"supplementary written msjlxerials may be needed^; In some cases, 
the communicating respocEsibii ity may even extend to preparing 
the supporting technical parts of whatever document emerges 
from the dec isionmaking process. . .: 

It is the responsibility of the staff which performed, the 
work to be available to the decisionmaker to help in understand- 
ing and - using the study. The staff "should also make a diftgent 
effort to find out'whethQt or not the study was useful. Lessons 
learned in this way can lead to better studies the next timie. 



^' The purpose of this appendix is to list some references 
that Should be useful to^those persons having limited exper-i- 
ence in conducting evaluations and analyses. The references 
are listed under several categories: basic disciplines, 
quantitative me^:hods, evaluation, and analysis. 

BASIC D ISCI PLINES " 
Baumol, .William J., Ec2r2omiG_Tneori:_a^ 

' ■ 3d ed.. Englewood Cliffs, N.J;, Prent ice-Hallv Inc., 1972. 

An excellent treatment of basic economic concepts apd . 
quantitative methods as they would apply to issues of ; 
resource allocation. 

Downs, Anthony, lnsi^§_§H££au c r ap^ . Boston, Little, Brown, ^ 
1967;" ^ ■ ,'■ ■ . ' ' 

• Very useful insights- concerning bureaucratic behavior 
and setting. • " •.. ^ • 

^Haveman, Robert H. and Jul ius --Margoris , eds ., Public Ex£end i- 
tuULand_Policx_Analxs^^^ Chicago, Markham Publishing 
CoTT 1970. 

■ ' Various aspects ot public expenditure economics are 
discussed. 'The economic bases of public . expenditures 

• are. developed in part I. Part II , with its emphasis on 
institutional considerations, is of special interest, as 
is part III wnich is concerned with ■ analytic problems- in 
policy analysis. The remaining parts survey the plan- 
ning ,, programing , budgeting , and system experience and. 
offer suggestions.. 

Rivlin , Alice M. , S^stemati^Thi^kin^ 

Washington, D.C., The Brookings Institution., 1971. 

A provocative series of .., essays on the issues involved 
in attempting to, solve the problems of society. 

Strunk, William and E. B. White/The Elements of. St^le. 
New York, Macmiilan, 1959. ; -■ 

A, very good writing style manual stressing .clear and 
concise . writing . . • 

■QUANf ITATIVE_riETHODS . ^ ' ' 

HilliV, Frederick S. and Gerald J ^ Lieberman , Introduction^ 

• to oper at i9ns_Re search. San Francisco, Ho Id en-Day, 
In5;T, 1967. ~ , ., : 
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Although a knowledge of mathematics is required/ the 
t^xt presents a comprehensive survey of the method's, 
..moc^^ techriicues that are used in analyses/ 

^Levin , ^Richard I . and Charles/A. Kirkpatrick, Qua ntitative . 
' Approaches to Management /' 2d ed . New "i^orlc, McGraw-Hill r 

■ An introduction to quantitative mabhadtS aHd. I^^BHRi qUBS • .. 

Moroney, M. Ji, Facts From Figures , Srd.ed; Baltimore, 
. Penguin Books , - Inc . , 195.6 . :^ 

• ' '■■■■■*.. \ • . "n ■ . ' 

■• /• A very r-eadable treatment of the use and misuse , of 
statist ical techniques ., .. ^ - 

Raif fa , Howard , Decision Analysis Introductory Lectures 

oh Choices Under Uncertainty , ' Read ing> Mass,'y Add isonr- 
Wesley Publishing Co. 1968 . 

A clear exposition of the process of./de terminihg bes t 
choices under uncertainty' and o,f^ considerations affecting 
^■'■'■'^ group decisions'. v . i • * • 

Tanur, Judith M. and pthers, eds • Statistics A Guide to 
t he Unknown . San Francisco , Holden-pay , Inc . , 1972 . 

Appl ica tions of statistics and {'probability are developed 
in a case method setting. . i . 

■■" . ' ■ ■ ■ i ■■ V ■ ■ ■ ■■ ■■ ■ 

Wohnacott Thomas H. and Ronald J.^. Wonnacott, Introductory 

Statistic s . 2d ed . l)Jew York, John Wiley & Sons> Inc./' 

1972. ^ \ ! ^ 



A- f airJL.y^r igorous;.^ but understandable "text of statistical 
-inference, including Bayesian-jm^hods and nonpar ametfic 



statistics . 



EVALUATION 

Hatry, Hairry P., Richard E. Winnie'/ and Donald M. -Fiskr 

Practical Program Evaluation /for State a nd Local Gov -' 
e rnment Officials . Washington-, D.G. y Urban- Institute ^ 
T973: ^. ~- \ 

A very useful primer on evaluation with. good examples, 
including an .excellent discussion of "comparison methods, 
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Isaac Stephen and William Michael , .Handbook^ih^Re sear ch 
and Ev aluation . San Diego, Knapp, ^9Jh 

A compendium of useful checklists, do's and don'ts, and . 
..summaries of important concepts and techniqyes for 
..'evaluation. ^ ." v . 

kiecken", Henry W. and Robert F. Boruch, eds., S ocial Experi - 
men tation; A Meth o d for Planni ng Vnd_Spcial Interven- 
tion . New York, Academic Press, 1974. 

An invaluable* guide—technically , ethically , and admin is- 

■ •tratively— in using experimental designs for evaluations. 
Excellent annotated bibliography on experiments. : 

Suchman ," : Edward A. , Evaluat ive Resear chi_^i.nc.iples and • 

P ractices JjT__PuEric"'si'F ^e and So cial Action Programs . 

■ New York , "..Russell Sage, 1967. 

' Considered to be a classic text in, evaluation . ^ Chapter 
IV, Categories of Evaluation, gives a- framework useful^ 
- in developing evaluative , questions. for a proposed study, 

. Weiss, carol, Evaluat ion_Re^^ 

program Effectivenes s . Englewood CI iff s , N .J .,. Prentice 

Hall,' 1972. 

• An excellent introductory text for evaluation. . - 

woiss Carol '^ed. , Rv^luat ing Action progra ms; Readings in 

197 2. ■ . ' ' 

Collection of articles dealing with basic concepts and 
' issues in evaluation,, especially for social programs. 

ANALY SIS ■ . '• ■ , " ,'■ ' 

Amacher^ Rvan C Robert D. Toll ison," and ThbWs D. Willett, 
^^^-^!^4 7^^/^om ic Approach to Public^ PoUcz. Ithaca, 
: .N.'Y:.:, Cornel University Press, 1976. 

* 'A collection of arti-^Ies dis^cussing'' the role , of econo- 

mists in public decisionmaking and -.applieatipns in a . 
range of programs, problems, and. issues. , ? ' ' 

Dolbeare',- Kenneth M., ed . , iuh}J^Jio]Ac^_^^ 

Hii'ls, Sage publications, 1975.. ■ „ . . 
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V A collection of recent .thoughts> on distinctions/betw 
• \ Social science research and. policy analysis irefated to - 
p'olitical f easibil ity and action , with extensive dis- 
cus s i o n o f app 1 i C a t i o,,n.s t o . c r ime c 6 nt r o 1 . 

Dor f man A Rpber t , ed . Measur ing Benefits of Governmental 

/Inve^stment , Washington , D.C. , The Brookings Institution/ 

■•/.■"1965V, ^- ■■ ' : - 

A seri^^s of cbntr ibuted papers concerned ^-with the appli- : 



, c.ationy of cost-benefit analysis. Wide ranging applica- 
tions are discussed . . ^ ^ 

Eng 1 i sh , J . ilor 1 ey , ed . / Cost-Effectiveness , New Yor k ; John: . 
Wiley & Sons, It^c.., 1968. . 



, A series of paper s cover ing the various aspects of cost- 

. . effectiven analysis, - 

Fisher.r Gene H. Cose Considerations:^ in Systems Analysis , 

New York, American Elsev ier Publishing Co . , Inc. , 1^71. 

•/;/.■;■. - • , , ■ '■./-.■■■ ^ ^ ^ , ^ 

. A^comprehensivie treatment of all relevant cost considera--: 
tSjjns that should . be- treatevi in analysis. Chapters 2^, 
2/ 3, and 6 are especially useful. . ' * 

Gregg / Phillip M; , ed . , Problems O'f Theory in Pol icy Analysis . 
Lexington , Massachusetts , Lexington Books, 1976. ' 

' An. excellent soi^rce of • recent thought on*, theoretical / 
. problems aad premises of ppiidy analysis . 

Hatry, Harry, Lbuis Blair, Donald Pisk, and; Wayne Kimmel,^^,: 
' • • Program Analysis for State and Local Governments . 
Washington , D.C. , Urban Institute / 1976 . 



,1 




* ,/ A practical / readable pr imer on analysis , including 
; the basic elements \of program analy 

■ / ■ ■ ■ ■' ■' ' ■•• ■'. " ■ ■ ■ . ■■ :. ■ ■ • ■ ■ ■ ' ' . ^ 

/(Hitch / Cha'r les - J . :ahd Roland N , McKean// The" E of ' 

' ' Defense, in the Nucle'ar Age . Canibr idg e , . Ha r v a r d Un i ve r.^- r/^ 
■ sity Presjs, 1960. / • 

V- ! Although the: application is military, a -classic discus-; 
sion of cost-effectiveness is found .in. chapters 7 ; 9 , ' 
lOV 11/ and- 12/ \ 

Nagql, Stuart S . V ed . , ; Pol icy Studies and the Social Sciences ^ 
Lexington, Massachusetts , Lexington Bo'oks , 1975. ^ ; . 
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' ;a collection of articles emphasizing the, contribution 
to policy studies which .can be, made by particular dis- 
' ,. ciplines, including sociology, psychology, anthropology, 
mathematics," law, and political science. , , 

Quade, E. S., Analysis for Public Decisio ns. New York, Ameri- 
can Elsevier Publishing Co., Inc., 1975. 

' a highly important' and readable book on all analytic 
aspects involved in formulating and implementing policy 
, decisions, . . . 



